Add test compliance audit template (spec+validation -> test code)#52
Merged
Alan-Jowett merged 3 commits intomicrosoft:mainfrom Mar 20, 2026
Merged
Conversation
Add a new template that audits test code against a validation plan and requirements document for test compliance drift. Detects unimplemented test cases (D11), missing acceptance criterion assertions (D12), and assertion mismatches (D13). New components: - Protocol: test-compliance-audit — 6-phase methodology (validation plan inventory, test code inventory, forward/backward traceability, classification, coverage summary) - Template: audit-test-compliance — consumes requirements + validation plan + test code, produces investigation-report Extended components: - Taxonomy: specification-drift — D11-D13 labels defined (previously reserved). Ranking criteria updated. No more reserved labels. - Scenarios gallery updated (moved from future to existing) This completes the audit trifecta: - audit-traceability: doc <-> doc <-> doc (D1-D7) - audit-code-compliance: spec <-> code (D8-D10) - audit-test-compliance: spec <-> test code (D11-D13) Closes microsoft#38 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new audit-test-compliance template and supporting taxonomy/protocol content to detect test compliance drift between requirements + validation plan vs automated test code (D11–D13).
Changes:
- Introduces
audit-test-compliancetemplate to audit validation-plan test cases against test code and report drift findings. - Adds
test-compliance-auditreasoning protocol (6 phases) for inventorying, tracing, and classifying drift. - Extends
specification-drifttaxonomy with concrete D11–D13 labels and updates docs/manifest to register the new components.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| templates/audit-test-compliance.md | New template for auditing test code vs validation plan + requirements using D11–D13 |
| taxonomies/specification-drift.md | Defines D11–D13 labels and updates drift ranking guidance |
| protocols/reasoning/test-compliance-audit.md | New 6-phase methodology for test compliance audits |
| manifest.yaml | Registers the new protocol and template |
| docs/scenarios.md | Promotes the scenario from “future” to current with richer metadata |
…everity - PARTIALLY IMPLEMENTED now maps to the correct drift type based on what's missing (criteria -> D12, assertions -> D13, stub -> D11) - Incomplete code context is INCONCLUSIVE, not D11 (same pattern as code-compliance audit) - D11 severity no longer references manual-only tests (those are excluded from findings by the protocol) - acceptance criterion -> acceptance criteria (plural) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Test stubs (empty body, skip annotation) are now D13 (assertions don't match because there are none) with a code location, not D11 (which implies no test function exists at all) - Orphaned tests referencing invalid TC-NNN/REQ-IDs are reported as observations in the coverage summary, not as D11-D13 findings (they don't fit the taxonomy since no valid TC-NNN is involved) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #38
Adds a new template that audits test code against a validation plan and requirements document for test compliance drift — gaps between what was planned for testing and what the automated tests actually verify. Completes the audit trifecta.
The Audit Trifecta
audit-traceabilityaudit-code-complianceaudit-test-complianceNew Drift Types (D11–D13)
New Components
protocols/reasoning/test-compliance-audit.md— 6-phase methodologytemplates/audit-test-compliance.md— consumes requirements + validation plan + test codeReused Components
specification-analystpersona,investigation-reportformat,anti-hallucination+self-verification+operational-constraintsguardrails,specification-drifttaxonomy (extended).Validation
tests/validate-manifest.pypasses